k8s - Kubernetes
- See README.md in git@github.com:mroverton1/private-devops.git
- Discuss - Forums
- https://github.com/DevOps-Nirvana/Universal-Kubernetes-Helm-Charts - DevOps - Nirvana
kubectl
Krew - kubectl plugin manager
- Krew - kubectl plugins1
kubectl krew update kubectl krew search k krew update; k krew upgrade s kubectl krew install ctx kubectl krew install ns
Learning
- https://github.com/kubernetes-up-and-running/examples - Kubernetes: Up and Running
Cut-n-Paste
# see krew above k top no k top po # troubleshooting kubectl get nodes -v=10 kubectl events -n flux-system --for OCIRepository/podinfo --watch kubectl config set-context --current --namespace=<namespace-name> journalctl -u kubelet kubectl config get-contexts # new env asdf plugin list all asdf plugin add kubectl asdf plugin add flux2 ... asdf list all <util> # get a list of available versions asdf install <util> <version> asdf set <util> <version> # set the version for this directory and below asdf current # list of active plugins kubectl version flux version k3s kubectl get nodes -o wide
Cluster Create / Configure
kubectl api-resources
- Arkade - The Open Source Kubernetes Marketplace
- https://rootlesscontaine.rs/ - rootless containers
- https://github.com/doitintl/zorya - GCP on Schedule
Workloads
Resources
resources: limits: cpu: 500m memory: 500Mi requests: cpu: 100m memory: 50Mi
- https://github.com/goharbor/harbor — Harbor is an open source trusted cloud native registry project that stores, signs, and scans content
- https://containrrr.dev/watchtower/ — Automatic image updates
Serverless
- https://github.com/openfaas/faas - https://www.openfaas.com/ Serverless functions
- https://github.com/fission/fission - serverless functions
- https://github.com/nuclio/nuclio - event and data processing
- https://github.com/kalmhq/kalm - Application Manager
Keel - automate updates
KubeApps from BitNami
- https://kubeapps.dev/ — deploy apps inside cluster
- https://kubeapps.dev/docs/latest/
kubectl get --namespace default \ secret kubeapps-operator-token \ -o go-template='{{.data.token | base64decode}}'
From docker-compose
Job and CronJob
Running a bash script inside a Job object
containers: - name: phylum-runner image: 262x09.dkr.ecr.us-west-2.amazonaws.com/backend:4.1.626 command: - bash - -c - | for y in $(seq 2021 2023); do for m in $(seq 1 12); do ./manage.py tenant_command run_report_phylum -s prod1 --year=$y --month=$m done done envFrom: - configMapRef: name: x-production-backend-conf
Network
Internal DNS for Services and Pods
# rabbitmq example BROKER_URL=\ amqp://<username>:<password>\ @<kubernetes-service-name>.<namespace>.svc.cluster.local:5672/<virtual-host>
- https://github.com/projectcontour/contour - ingress controller
Security / Access
- https://github.com/falcosecurity/falco - runtime security
- https://github.com/aquasecurity/kube-hunter - looks for security weakness (old)
- https://github.com/open-policy-agent/gatekeeper - policy controller
- https://github.com/kyverno/kyverno - policy manager
- https://github.com/bridgecrewio/checkov - config checker
- https://github.com/yannh/kubeconform - manifest validator
- https://github.com/kubescape/kubescape - security scan
- https://github.com/openshift/origin - conformance test suite for openshift
- https://github.com/project-copacetic/copacetic - directly patch container image vulnerabilities
- https://github.com/tenable/terrascan - detect compliance and security violations
Trivy
- https://github.com/aquasecurity/trivy - secuity scanner
- https://github.com/aquasecurity/trivy - image scanner
Kyverno - Policy Engine
- https://kyverno.io/ - Policy Manager
- Signing Bin - sigstore
- Starboard Operator - vulnerability scanner
- https://www.pomerium.com/docs - id aware proxy
Storage
- https://github.com/wagoodman/dive - see what is in container
Namespace
- vcluster - in a namespace
Backup procedure
# stop services for db backup kubectl scale deployment <deployment-name> --replicas=0 -n <namespace> kubectl patch cronjob <cronjob-name> -n <namespace> -p '{"spec": {"suspend": true}}' kubectl get pods -n <namespace> # backup kubectl scale deployment <deployment-name> --replicas=<original-replicas> -n <namespace> kubectl patch cronjob <cronjob-name> -n <namespace> -p '{"spec": {"suspend": false}}' # check kubectl get pods -n <namespace> kubectl get deployments -n <namespace>
Monitor and Logging
- https://github.com/google/cadvisor then add prometheus
- https://github.com/thanos-io/thanos - prometheus setup. thanos getting started
Logging
- https://github.com/stern/stern — tail logs (old)
https://github.com/boz/kail - tails the logs
# Include/Exclulde namespace kail -n airflow kail --ignore-ns=airflow
- https://middleware.io/ - commercial monitoring
Clients
kubectl
export KUBECONFIG=$PWD/.kube/cp-kdev-k3s.yaml kubectl ctx kubectl ns kubectl auth can-i get pods kubectl describe pod -n x-production > mro/describe-pods-prod.yml kubectl logs podname -c containername --previous kubectl config view --minify kubectl exec -it x-production-api-xxx -- /bin/bash
Tools
Connect to AI
- https://github.com/xeol-io/kubegpt - links to OpenAI
- https://github.com/k8sgpt-ai/k8sgpt - scan and diagnos
- https://github.com/FairwindsOps/pluto - tool to help with k8s objects upgrade
https://codeberg.org/hjacobs/kube-ops-view - web ascii dashboard
- hjacobs - other useful tools?
- Zora - Help with k8s best practices
- https://www.parca.dev/ - profiling
Troubleshoot Pending Pods
Sonobuoy
sonobuoy run --wait results=$(sonobuoy retrieve) sonobuoy results $results sonobuoy delete --wait # during sonobuoy status sonobuoy logs
- Elemental enable centralized, full cloud-native OS management with Kubernetes
- https://carvel.dev/ - kapp controller
- https://www.datree.io/ - datree
- https://keel.sh/ - Operator to automate deployment updates
- https://github.com/wagoodman/dive - see inside containers
- https://github.com/kubernetes-sigs/kubespray - deploy prod ready clusters. Docs: https://kubespray.io/#/
- https://github.com/weaveworks/scope - weaveworks
- https://kops.sigs.k8s.io/ - get a production grade Kubernetes cluster up and running
- https://kro.run/ - resource orchestrator
- https://keptn.sh/stable/ - deployments
- DZone - Best Kubernetes Tools
Kalm - k8s setup
- https://docs.kalm.dev/ - Kalm makes it easier to setup a working Kubernetes
- https://github.com/kalmhq/kalm - last commit in Apr21
Portainer on K8S
kubectl create namespace portainer helm repo add portainer https://portainer.github.io/k8s/ helm repo update helm upgrade -i -n portainer portainer portainer/portainer export NODE_PORT=$(kubectl get --namespace portainer -o jsonpath="{.spec.ports[1].nodePort}" services portainer) export NODE_IP=$(kubectl get nodes --namespace portainer -o jsonpath="{.items[0].status.addresses[0].address}") echo https://$NODE_IP:$NODE_PORT
Clean up replica sets
kubectl get rs --all-namespaces | awk '$3 == 0 {print $2, $1}' | while read rs namespace; do kubectl delete rs $rs -n $namespace done
Spacelift
Load Testing
Kubernetes As Service
Policy Manager
- https://kyverno.io/ - Native policy manager
Cost Cutting
- Perfectscale for cost optimization
Karpenter
10.4.18.35:8443->10.4.16.236:40672
Get Karpenter working